Add the mean EnKF soil increment to the deterministic member #3295

ClaraDraper-NOAA · 2025-02-04T18:16:33Z

Description

Add mean EnKF soil increment to the deterministic member. Includes new regriding script and executable (in GDASApp) to perform masked bi-linear interpolation of the soil increments.

Resolves 2773

Type of change

Bug fix (fixes something broken)
[ x] New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules?
- [ YES] UFS-utils

OUTDATED:
Exercizing the new soil analysis feature requires updating UFS_UTILS to at least commit e718003

UPDATE Feb 5:
After responding to reviews, requires updating to UFS_UTILS PR 1022

How has this been tested?

Tested on hera:
C96_atm3DVar, C96C48_hybatmaerosnowDA, and the default test ran succesfully.
Confirmed new functionality works as expected.
Confirmed that the commits do not change output from cycling the DA if the the new feature is not selected.

Checklist

[ x] Any dependent changes have been merged and published
[x ] My code follows the style guidelines of this project
[ x] I have performed a self-review of my own code
[x ] I have commented my code, particularly in hard-to-understand areas
[x ] I have documented my code, including function, input, and output descriptions
[ x] My changes generate no new warnings
[ x] New and existing tests pass with my changes
This change is covered by an existing CI test or a new one has been added
[x ] Any new scripts have been added to the .github/CODEOWNERS file with owners
[x ] I have made corresponding changes to the system documentation if necessary

NOTE:

Exercizing the new feature requires some additional fix files, currently at $TMP_FIX_FILES in ush/regrid_gsiSfcIncr_to_tile.sh . These will need to be added to a fix directory, if this PR is accepted as is (note: these files could also be generated on the fly)
Exercizing the new feature requires jobs/rocoto/sfcanl.sh and sfc.sh to load the ufsda modules in place of the fv3gfs modules. My understanding is that the modules will eventually be unified, but for now I've added comments to the relevant scripts noting this.

i) write out the ensemble mean reanalysis, and apply it to the deterministic member. ii) replace UFS_UTILS regridding of Gaussian soil analysis to model grid with an external ESMF-based program (with appropriate land masking) Requires: sorc/gsi_enkf.fd develop after Tue Nov 12. Latest works. sorc/gdas.cd/sorc/da-utils PR: NOAA-EMC/DA-utils#5 sorc/ufs_utils.fd from PR: ufs-community/UFS_UTILS#1010

…s) case

…sis.

github-advanced-security

ShellCheck found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

CoryMartin-NOAA

first pass

env/GAEAC5.env

jobs/JGLOBAL_ATMOS_SFCANL

parm/config/gfs/config.sfcanl

scripts/exgdas_enkf_sfc.sh

scripts/exglobal_atmos_sfcanl.sh

ush/regrid_gsiSfcIncr_to_tile.sh

aerorahul

the changes looks good. seems like some checks are failing. I tried to recommend a couple updates that might help.

ush/regrid_gsiSfcIncr_to_tile.sh

Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

RussTreadon-NOAA · 2025-02-25T20:40:07Z

WCOSS2 note

g-w CI does not run C96C48_hybatmaerosnowDA on WCOSS2. GFS v17 will exercise updates from this PR. GFS v17 runs on WCOSS2. This is problematic. The gdas and enkfgdas snowanl jobs currently fail on Cactus.

ClaraDraper-NOAA:feature/soilanal_det_clean at 7fd68e8 was installed on Cactus. g-w CI C96C48_hybatmaerosnowDA was set up and launched. As expected the 00Z snowanl jobs died

russ.treadon@clogin08:/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3295> rocotostat -d C96C48_hybatmaerosnowDA_pr3295.db -w C96C48_hybatmaerosnowDA_pr3295.xml |grep DEAD
202112210000            gdas_snowanl                   181634255                DEAD                   1         2          59.0
202112210000       enkfgdas_esnowanl                   181634258                DEAD                   1         2          94.0

The reason for the failure is the same in each job

ImportError: /opt/cray/pe/lib64/libmpifort_intel.so.12: undefined symbol: MPIX_Enqueue_send

Making the following change to ush/load_ufsda_modules.sh

@@ -41,6 +41,7 @@ case "${MACHINE_ID}" in
       # TODO: Add path to GDASApp libraries and cray-mpich as temporary patches
       # TODO: Remove LD_LIBRARY_PATH lines as soon as permanent solutions are available
       export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HOMEgfs}/sorc/gdas.cd/build/lib"
+      export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/cray/pe/mpich/8.1.19/ofi/intel/19.0/lib"
     fi
     module load "${MODS}/${MACHINE_ID}"
     ncdump=$( command -v ncdump )

allows both failed jobs to run to completion.

202112210000            gdas_snowanl                   181638536           SUCCEEDED                   0         1         189.0
202112210000       enkfgdas_esnowanl                   181638535           SUCCEEDED                   0         1         170.0

Reached out to NCO, GDIT, and the library team to see if they know why we need to add the line

export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/cray/pe/mpich/8.1.19/ofi/intel/19.0/lib"

back to load_ufsda_modules.sh. We were able, at least temporarily, to remove this line after RFC 13644 was implemented 5-6 February 2025.

emcbot · 2025-02-26T00:44:31Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3295/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_4fd0ec17/logs/2021122018/gfs_metpg2o1.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-02-26T00:44:38Z

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3295/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_4fd0ec17

emcbot · 2025-02-26T01:57:52Z

CI Failed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3295


Experiment C96mx100_S2S_4fd0ec17 Completed 1 Cycles: *SUCCESS* at Tue Feb 25 22:41:42 UTC 2025
Experiment C48_ATM_4fd0ec17 Completed 1 Cycles: *SUCCESS* at Tue Feb 25 23:18:09 UTC 2025
Experiment C48mx500_hybAOWCDA_4fd0ec17 Completed 2 Cycles: *SUCCESS* at Tue Feb 25 23:36:26 UTC 2025
Experiment C96C48_hybatmDA_4fd0ec17 Completed 3 Cycles: *SUCCESS* at Wed Feb 26 00:38:06 UTC 2025
Experiment C96C48_hybatmaerosnowDA_4fd0ec17 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Feb 26 00:44:13 UTC 2025
Experiment C96C48_hybatmaerosnowDA_4fd0ec17 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3295/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_4fd0ec17/logs/2021122018/gfs_metpg2o1.log
Experiment C96_atm3DVar_4fd0ec17 Completed 3 Cycles: *SUCCESS* at Wed Feb 26 00:56:21 UTC 2025
Experiment C48_S2SW_4fd0ec17 Completed 1 Cycles: *SUCCESS* at Wed Feb 26 01:02:29 UTC 2025
Experiment C48_S2SWA_gefs_4fd0ec17 Completed 1 Cycles: *SUCCESS* at Wed Feb 26 01:09:19 UTC 2025
Experiment C96C48_ufs_hybatmDA_4fd0ec17 Completed 3 Cycles: *SUCCESS* at Wed Feb 26 01:45:25 UTC 2025
Experiment C48mx500_3DVarAOWCDA_4fd0ec17 Completed 2 Cycles: *SUCCESS* at Wed Feb 26 01:57:37 UTC 2025

aerorahul · 2025-02-26T13:39:01Z

@ClaraDraper-NOAA
I discussed w/ @malloryprow and we found that the met datasets required for the failed job were deleted to free up space on Hera.
I think this PR is good to merge.

emcbot · 2025-02-26T15:23:09Z

CI Tests set up to run in /lfs/h2/emc/ptmp/emc.global/PR/PR_3295/RUNTESTS on WCOSS

DavidHuber-NOAA · 2025-02-27T14:18:46Z

@aerorahul @ClaraDraper-NOAA There are some permissions issues with some of the prepbufr files used for verification in the metpg2o1 job. The RDHPCS issue RDHPCS#2025022654000078 was opened to sort this out. In the interim, should we temporarily disable the METplus jobs on Hera?

aerorahul · 2025-02-27T14:20:35Z

@DavidHuber-NOAA
Yes. I think that would be fine.
Meanwhile, lets merge this PR now that WCOSS2 has passed and Hera issues are unrelated and known.

ClaraDraper-NOAA added 16 commits December 18, 2024 23:41

Adding missing ush file

f9e9a49

Clean up calculation of forecast times required for regridding.

8120fb6

Merge branch 'develop' into feature/soilanal_det

0f150a9

Merge branch 'develop' into feature/soilanal_det

ea61f31

Added dependency on enkf for gdas_sfcanl, for GSI_SOILANAL case

1f3c8df

Tidy up timing of adding the sfc increments. Update config and xml.

0bf0a90

Fixed bug in FHR calc in regrid, added soil analysis to non-IAU (atmo…

e478210

…s) case

Added LSOIL_INC to config files. generalized regridding namelist input.

3856b1f

workaround to ease merging.

d8e945f

Merge branch 'develop' into feature/soilanal_det_clean

45c6cc1

Added back in env changes after merge.

86e0deb

Tidy up.

59e82b7

Merge branch 'develop' into feature/soilanal_det_clean

3d5b6ee

Updated UFS_UTILS to include latest update, needed for the soil analy…

b02e8af

…sis.

Updated code owners.

7ffee90

ClaraDraper-NOAA requested review from GeorgeGayno-NOAA, CoryMartin-NOAA, RussTreadon-NOAA, CatherineThomas-NOAA, WalterKolczynski-NOAA, KateFriedman-NOAA, DavidHuber-NOAA and aerorahul as code owners February 4, 2025 18:16

github-advanced-security bot found potential problems Feb 4, 2025

View reviewed changes

coding norms.

5cccc3f

CoryMartin-NOAA reviewed Feb 4, 2025

View reviewed changes

aerorahul reviewed Feb 4, 2025

View reviewed changes

ush/regrid_gsiSfcIncr_to_tile.sh Outdated Show resolved Hide resolved

ush/regrid_gsiSfcIncr_to_tile.sh Outdated Show resolved Hide resolved

ClaraDraper-NOAA and others added 2 commits February 4, 2025 15:12

Update parm/config/gfs/config.sfcanl

04e542e

Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

Update scripts/exgdas_enkf_sfc.sh

fae30c6

Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

aerorahul added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Feb 25, 2025

emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Feb 25, 2025

emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Feb 25, 2025

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Feb 26, 2025

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Feb 26, 2025

This was referenced Feb 26, 2025

snowanl fails on WCOSS2 with undefined mpi reference #3395

Closed

Add mpich/8.1.19 to the WCOSS2 LD_LIBRARY_PATH for GDASApp jobs #3396

Merged

Merge branch 'develop' into feature/soilanal_det_clean

056e9b3

emcbot added the CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS label Feb 26, 2025

KateFriedman-NOAA removed the CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS label Feb 26, 2025

emcbot added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Feb 26, 2025

RussTreadon-NOAA mentioned this pull request Feb 27, 2025

Enable GDASApp g-w CI cases to run on wcoss2 #3399

Open

6 tasks

WalterKolczynski-NOAA added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Feb 27, 2025

aerorahul merged commit 25778ff into NOAA-EMC:develop Feb 27, 2025
5 checks passed

RussTreadon-NOAA mentioned this pull request Feb 27, 2025

correct logic for upper loop limit in chem_diagb NOAA-EMC/GDASApp#1515

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the mean EnKF soil increment to the deterministic member #3295

Add the mean EnKF soil increment to the deterministic member #3295

ClaraDraper-NOAA commented Feb 4, 2025 •

edited

Loading

github-advanced-security bot left a comment

CoryMartin-NOAA left a comment

aerorahul left a comment

RussTreadon-NOAA commented Feb 25, 2025

emcbot commented Feb 26, 2025

emcbot commented Feb 26, 2025

emcbot commented Feb 26, 2025

aerorahul commented Feb 26, 2025

emcbot commented Feb 26, 2025

DavidHuber-NOAA commented Feb 27, 2025

aerorahul commented Feb 27, 2025

Add the mean EnKF soil increment to the deterministic member #3295

Add the mean EnKF soil increment to the deterministic member #3295

Conversation

ClaraDraper-NOAA commented Feb 4, 2025 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

github-advanced-security bot left a comment

Choose a reason for hiding this comment

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

RussTreadon-NOAA commented Feb 25, 2025

emcbot commented Feb 26, 2025

emcbot commented Feb 26, 2025

emcbot commented Feb 26, 2025

aerorahul commented Feb 26, 2025

emcbot commented Feb 26, 2025

DavidHuber-NOAA commented Feb 27, 2025

aerorahul commented Feb 27, 2025

ClaraDraper-NOAA commented Feb 4, 2025 •

edited

Loading